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ABSTRACT 



A meta-analysis considered the impact of various methods of 
improving public communication skills on critical thinking. Manuscripts were 
obtained by searching the available electronic databases (COMINDEX, 
Dissertation Abstracts, ERIC, Index to Communication Journals, PSYCHLIT) , 
various bibliographies available on the topic, as well as the reference 
section of manuscripts obtained. Results indicated that communication 
instruction improves the critical thinking ability of the participants 
(longitudinal designs r = .176, cross-sectional designs r = .196). Forensic 
participation demonstrated the largest positive impact on critical thinking 
improvement, but all communication skill experiences demonstrate significant 
improvement. The cumulative evidence indicated that communication skill 
instruction generates, using the Binomial Effect Size Display (BESD) , a 44% 
increase in critical thinking ability. This summary of available research 
provides documentation supporting the claims of departments and educators for 
the viability of communication skill instruction as a means of improving 
critical thinking. These results provide important evidence to support the 
maintenance of forensics programs in an era of increased educational 
accountability, downsizing, and budgetary cutbacks. (Contains 5 tables of 
data and 39 references.) (Author/NKA) 
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ABSTRACT 

Measuring the Impact of Forensics and 
Communication Education on Critical Thinking: 

A Meta-analytic Summary 

This meta-analysis considers the impact of various methods 
of improving public communication skills on critical thinking. 

The results indicate that communication instruction improves the 
critical thinking ability of the participants (longitudinal 
designs r = .176, cross-sectional designs r = .196). Forensic 
participation demonstrated the largest positive impact on 
critical thinking improvement, but all communication skill 
experiences demonstrate significant improvement. The cumulative 
evidence indicates that communication skill instruction 
generates, using the Binomial Effect Size Display (BESD) a 44% 
increase in critical thinking ability. This summary of available 
research provides documentation supporting the claims of 
departments and educators for the viability of communication 
skill instruction as a means of improving critical thinking. 

These results provide important evidence to support the 
maintenance of forensics programs in an era of increased 
educational accountability , downsizing, and budgetary cutbacks. 
Key Words : Critical Thinking, Meta-analysis, Forensics, 

Communication Education, Public Speaking, Mock trial. Individual 
Events . 
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One of the central arguments for instruction in public 
speaking, argumentation, persuasion, and debate is the belief 
that such training improves the critical thinking ability of the 
participants (for a complete discussion of the range goals of 
forensics, that would apply to other public communication 
instruction, see Hunt, 1994) . Hill (1993) points out that 
demands for educational accountability require that forensics (as 
well as other communication courses) document to what degree 
those activities meet educational goals. The increasing need for 
university programs to provide "proof of service" creates a need 
for communication courses and activities to document the 
improvements generated. Such claims can be supported with 
testimonials, antedoctal stories, and examples from participants. 
However, systematic and more objective indications of improvement 
would provide an additional level of support more persuasive to 
some critics . 

Garside (1996) surveyed the impact of group discussion 
teaching strategies on the development of critical thinking 
skills and concluded that: 

the literature suggests at least four defining aspects 
of thinking that make is critical: (a) thinking that 

is clear, precise, accurate, relevant, logical, and 
consistent; (b) thinking that reflects a controlled 
sense of skepticism or disbelief of any assertion, 
claim, or conclusion until sufficient evidence and 







reasoning is provided to conclusively support it; (c) 
thinking that takes stock of existing information and 
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identifies holes and weaknesses, thereby certifying 
what we know or don't know; and (d) thinking that is 
free from bias, prejudice, and one-sidedness of thought 
(p. 215) 

Instructors of public speaking, discussion, and argumentation as 
well as those advocating participation in competitive forensics, 
have argued that such activities improve critical thinking. 
Unbridled by the limitations found within the traditional 
lecture-oriented classroom situation, participants must learn to 
invent, organize, and articulate thoughts subject to scrutiny by 
others. The student learns not only how to generate higher 
quality thinking but learns how to critique the arguments and 
conclusions of others. The process of participation as a 
communicator originating the argument and as a critical consumer 
provides a "hands on" experience that should improve critical 
thinking ability. 

Assuming that such improvement occurs, the conclusion is 
that the educational advantages of a person able to both generate 
and critique conclusions provides the basis for quality 
participation in the society and the academic environment. The 
conclusion offered is simple, participation in forms of public 
communication involved in public speaking, forensics (debate. 
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mock trial, individual events), and argumentation require 
attention to argument and counterargument. 

Previous Research 

There exists a number of investigations comparing the impact 
of various communication skill experiences on critical thinking 
(Colbert, 1995; Hill, 1993) . Available research summaries 
provide at best a mixed set of support for the argument that 
experience with communication skill exercises, particularly 
forensics, provides a measurable improvement in the area of 
critical thinking (Follert & Colbert, 1983). 

The typical investigation in this area takes some group of 
college students in a course (Allen, Berkowitz, & Louden, 1995; 
Hurst, 1962; Johnson, 1942; Ness, 1967) or a group of forensics 
participants (Allen, Berkowitz, & Louden, 1995; Bursack, 1961; 
Colbert, 1995) and compares the change over time in critical 
thinking or compares different groups. The concern of educators 
has prompted a series of investigations over the past fifty years 
exploring the impact of communication skill experiences on 
critical thinking. 

One issue not considered is the number of unpublished 
reports of data that exist. A search of computer indexes 
( Comlndex ) and the published literature finds relatively few data 
sets in existence. However, a number of unpublished doctoral 
dissertations (Beckman, 1955; Cross, 1971; Frank, 1964; Hurst, 
1962; Jackson, 1961; Ness, 1967; Tame, 1958) and unpublished 



master's theses (Baumgartner, 1965; Beihl, 1963; Bursack, 1961; 
Smith, 1942) exist. The inclusion or consideration of this set 
of information expands the available data base considerably. 

The current data base is sufficient to justify the use of 
meta-analysis as a means of summarization. The next section of 
the paper considers various criticisms of the existing 
literature. If all the available research employed fundamentally 
flawed designs or measurement, than any proposed summary of the 
research is meaningless. It is only after the literature is 
accepted as relevant that any summary becomes useful. 

Critiques of Existing Research 

The largest single criticisms of research examining the 
impact of communication skills training on improvements in 
critical thinking usually consider two issues: (a) Measurement 

of critical thinking, and (b) Selection of samples used for 
analysis . 

The Watson-Glaser tests (in all it's forms) represents the 
dominant form of measurement for critical thinking (for an 
example of the test see Watson & Glaser, 1951) . The paper and 
pencil objective test uses a multiple choice format. The 
questions involve tests of five different critical thinking 
skills (inference, recognition of assumptions, deduction, 
interpretation, and evaluation of arguments) . The test format is 
usually the use of an example and then some test of an 
implication that follows. The question is whether the person 
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can, using the accepted rules of inference, understand the 
permissible conclusions, inferences that one can make from the 
example or available data. 

The Watson-Glaser Test (1951) scores each answer as correct 
or incorrect. The methodological issue is whether one can 
measure critical thinking using an objective test and whether an 
objective test completely captures the domain of critical 
thinking. The test has a history of adequate reliability, the 
question is one of validity. Is the use of a paper and pencil 
test a "valid" measure of the concept under study. The answer to 
this depends on how the term "critical thinking" is applied. The 
Watson-Glaser test measures the ability of persons to follow the 
"rules" involved in various forms of reasoning. To the extent 
that one accepts the underlying rules, the test is valid. 
Considerable discussion exists about the adequacy of this measure 
and the underlying conceptual variable (see Greenstreet, 1993; 
Hill, 1993) . 

The issue of the Watson-Glaser test and the inadequacy of 
the available measurement remains a serious one and cannot be 
resolved by this meta-analysis in a satisfying manner unless a 
sufficient body of alternative research exists using other 
measures of critical thinking. But even then, the alternative 
measures, if they share the same conceptualization would only 
provide convergent validity. 
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The second issue raised deals with the use of forensics 
participants to measure critical thinking improvement or the 
impact of forensics participation on critical thinking (this 
could be extended to other communication skill experiences as 
well) . The argument is that forensic participants are self- 
selected, and the choice to participate in competitive forensics 
might be related to higher levels of existing critical thinking. 
Basically, the comparisons of forensic participants to 
nonforensic samples is not a fair comparison because of the bias 
in self-selection . The argument is that any estimate of the 
impact of forensic participation must consider or remove the 
impact of this sample bias. 

The argument mathematically is rather limited. The argument 
only applies to the use of cross-sectional designs. Longitudinal 
designs would be biased against demonstrating significant gains 
in forensic participants if this argument is true. The reason 
for that is the "ceiling effect." The ceiling effect is the term 
used to note that a group cannot improve very greatly if the 
score is already at the top of the scale. If the mean critical 
thinking score for forensic competitors is higher than the 
general population, an improvement score should be lower for 
forensic participants, the forensic participants simply have less 
room to improve than nonf orensicators . 

If the impact of the effect exists, the pattern of results 
should demonstrate a higher score for forensics participants in 




9 



9 



cross-sectional designs. The change score in longitudinal 
designs should favor forensic participants. Again, this 
assessment can be made if there exist a variety of designs to 
make this comparison. 

The arguments surrounding the validity of the existing 
designs rather than being accepted as fatal flaws, are flaws that 
potentially become the focus of assessment. The advantage of 
meta-analysis as a literature summarization technique is that the 
arguments about measurement and design become the sources and 
reason for conducting the analysis rather than a rejection of the 
conclusions . 

Justification for Using Meta-analysis (Again) 

A previous meta-analysis (Follert & Colbert, 1983) conclude 
that "the results of this analysis cast substantial doubt on the 
claimed relationship between debate training and critical 
thinking skill improvement" (p. 2). The conclusion derived from 
five studies indicates no significant correlation exists between 
participation in debate and subsequent critical thinking gains. 
The meta-analysis form Follert and Colbert used generated a set 
of 47 individual comparisons. Of the 47, 28 demonstrated a gain 
in critical thinking for debate while 19 did not. The conclusion 
they offer is that there is no consistent body of evidence for a 
gain in critical thinking skill. 

The previous meta-analysis was conducted using a 
technique and did a binomial test of the results of individual 
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significance tests. Unfortunately, the binomial test fails to 
consider the impact of Type II error and treats each trial as a 
separate outcome. The result is that the failure to achieve 
significance does not consider the impact of the separate 
incidence of Type II error for each sample. This form of meta- 
analysis has been eclipsed by the development of more 
sophisticated statistical forms of meta-analysis (particularly 
variance-centered forms, Bangert-Drowns , 1986. 

The problem with the previous form was that the results of 
the previous analysis do not establish an average effect, instead 
the effort is made to create a common metric and then look for 
consistency among the results of individual significance tests. 
Rather than averaging effects, the analysis is conducted at the 
level of individual units and a vote-counting method still 
employed . 

Methods 

Literature Search 

Manuscripts were obtained by searching the available 
electronic data bases (COMINDEX, DISSERTATION ABSTRACTS, ERIC, 
INDEX TO COMMUNICATION JOURNALS, PSYCHLIT) , various 
bibliographies available on the topic (Colbert, 1995; Follert & 
Colbert, 1983; Hill, 1993) as well as the reference section of 
manuscripts obtained. 

To be included in the analysis a manuscript had to meet the 
following conditions : 
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(a) contain quantitative data; 

(b) some type of communication skill improvement exercise 
(this could be in the form of a course or participation 
in competitive forensics) ; 

(c) some method of assessing critical thinking skill 
improvement (either using a cross-sectional or 
longitudinal design) . 

Coding of Study Features 

Study features were coded as potential sources of moderating 
influences. If the average results of the investigations are 
variable, then the search for moderator variables (particularly 
design differences) could serve as an explanation for such 
inconsistencies . 

Type of Research Design 

The investigations included in this analysis used two types 
of research designs to estimate the impact of communication skill 
training on critical thinking improvement: (a) Longitudinal, and 

(b) Cross-sectional designs. Some studies used both types of 
designs and contributed to both sets of analyses. A longitudinal 
design is one where there are multiple measurements on the same 
set of participants. The measure of the effect is a comparison 
of the scores from the first time period (pre-test) to the scores 
at a later time period, after exposure to the communication 
skills material (post-test) . The assumption is that improvement 
is based on the participation in the training program. The 
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change is the amount of improvement demonstrated between the two 
measurements . 

The second type of design, cross-sectional, is a static 
design (invariably a post-test comparison) between one group 
participating in some type of communication skills training 
versus some group that did not. The design represents a classic 
quasi-experimental design of an experimental and control group. 
The assumption is that each group started at the same level and 
the longitudinal comparison illustrates the impact of the 
differential experiences for the two groups. 

Type of Communication Skill Experience 

The type of communication skill improvement approach 
differed both between and within investigations. The experiences 
involved competitive forensic participation as well as courses 
designed to improve the skill of the student in accomplishing 
some communication task. The longitudinal experience was coded 
into the following categories: (a) public speaking class, (b) 

argumentation, debate, and discussion class and (c) competitive 
forensics participation (this includes debate, discussion, 
individual events, mock trial, or other types of participative 
competitive events) . The division represents the type of 
experiences available to the participant. The coding for 
individual studies is found in Table 2. 

The cross-sectional comparisons were divided into three 
types of comparisons: (a) communication skill activity 
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(forensics, public speaking class, or argumentation class) to a 
control group, (b) a public speaking or argumentation class to 
forensics, and (c) an argumentation class (or "special" public 
speaking class that included a section on argumentation and 
debate) compared to a public speaking class. 

The divisions between types of courses or activities based 
on content provides some type of assessment about whether a 
particular type of content would most improve critical thinking 
skills. Competitive forensics is often given academic credit and 
the provision for such credit is based on critical thinking 
improvement as well as skill at public presentation. 

Demonstrating the relative size of the improvements offered in 
critical thinking for each type of activity provides educators 
with information on what approach (es) can be expected to generate 
particular outcomes. 

Statistical Analysis 

The data were analyzed using the variance-centered form of 
meta-analysis developed by Hunter and Schmidt (1990) reflected in 
the computer programs META-COR (Hamilton & Hunter, 1990) and 

ATTEN (Hamilton & Hunter, 1991). 1 The technique takes the 
statistical information from investigations and converts the 
information to a common metric. The form recommended is the 
correlation coefficient. The estimates for each study are 
statistically transformed into a correlation. The conversion 
creates a common metric from each investigation 
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Transformed data are corrected for a variety of statistical 
artifacts and possible design limitations. Correction to the 
data for attenuated measurement or restriction in range is 
possible. This correction can consider the impact of selection 
artifacts that reflect the possibility of self-selection 
artifacts for debaters. The impact of both sets of conditions is 
to systematically underestimate the size of the effect (Allen, 
Hunter, & Donohue, 1989; Hunter & Schmidt, 1990) . So these 
corrections serve to increase the size of the existing effect, 
not reduce it. The average effect is therefore slightly larger 
after the corrections and the estimate of the variance larger as 
well. Each table provides the corrected correlations (r') as 
well as the uncorrected correlations (r) . 

The transformed data are then averaged using a formula that 
weights each correlation by the corresponding sample size. The 
reason that a weighted average is used is that the assumption is 
that larger samples provide more accurate estimates of the 
population effect than do small samples. The level of type II 
error is less with larger sample sizes. A study using a small 
sample size has a larger amount of sampling error and is 
therefore a less accurate estimate of the population parameter. 
Weighting by sample size is a recognition of this variability in 
the accuracy of the estimate. A moderator variable produces 
subsets the comparison of the mean effect of each subset 
demonstrates differences between the groups. 
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No data appears more than once in any particular subgroup 
although data sets are used in multiple subgroups. This creates 
a problem of the data lacking statistical independence. A Monte 
Carlo test of the impact of lack of independence demonstrates 
however, that estimates of both the mean and standard deviation 
are unaffected by this problem (Tracz, 1985) . 

Results 

Overall 

The comparisons in the overall analysis (as well as 
moderator analysis) considers the two basic types of designs 
(longitudinal and cross-sectional) separately. The results of 
the longitudinal designs illustrate that communication skill 
exercises improve critical thinking ( average r = .176, var . = 
.010, k - 17, N = 2657. 95% Confidence Interval [C.I.] ± .124) . 
Cross-sectional designs find that a communication skill exercise 
improves critical thinking ( average r = .196, var . = .028, k = 

13, N = 2395, 95% C.I. ± .291). 

A comparison of the difference in the size of the 
correlations demonstrates that while the cross-sectional design 
is larger than the longitudinal design (d = .105), the difference 
is relatively small. The most important consideration is that 
the effects are both positive, indicating that critical thinking 
improved when either using a longitudinal or cross-sectional 
comparison . 
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Type of Critical Thinking Measurement Instrument 

The studies fell into two basic groups: (a) studies that 

used some version of the Watson-Glaser Critical Thinking 
Appraisal (1951), or (b) some other measure of critical thinking. 
Table 1 provides a list of the measures used in each 
investigation. This analysis considers the longitudinal and 
cross-sectional data separately. 

Analysis comparing those studies using a longitudinal design 
demonstrates that the studies using Watson-Glaser measure for 
critical thinking ( average r = .160, var . - .010, k = 12, N = 

1875, 95% C.I. ± .124) showed improvement. The studies using 
some other measure observed a positive improvement (average r = 
.215, var. = .007, k = 5, N = 782, 95% C.I. ± .150). The 
comparison of these two average effects demonstrates a larger 
average for nonWatson-Glaser measures (d = .80). If an argument 
is made about that scale for this data is that it underestimates 
the real effect. 

The results using a cross sectional comparison of a 
communication skills experience to a control group were divided 
on the basis of measurement. The Watson-Glaser group 
demonstrated improvement ( average r = .192, var . = .029, k = 10, 

N = 2105, 95% C.I. ± .303). The group of investigations using 

other methods of measurement observed an average correlation that 
was positive ( average r = .226, var . = .013, k = 3, N = 290, 95% 
C.I. ± .123) . A comparison of the two mean effects demonstrates 
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the same pattern as the longitudinal data, the Wat son-Glaser 
measure demonstrates a smaller effect (d = .31). 

The results indicate that the Watson-Glaser method of 
measuring critical thinking improvement produces smaller effects 
than other measures. However, the effect is consistent in 
direction, if not magnitude, to all the other measures. The 
evidence suggests the possibility of differential validity for 
scales but the consistent pattern indicates convergent validity 
for the measures. 

Type of Communication Skill Experience 

The longitudinal designs contained examined different types 
of skill experiences used in this analysis: (a) public speaking 

class, (b) argumentation class, and (c) competitive 
debate/forensics. The longitudinal design makes it possible to 
measure the improvement for each condition separately and compare 
effect sizes. 

Participation in a public speaking improves the critical 
thinking skill of students ( average r = .145, var . = .066, _k = 6, 

N = 531, 95% C. I . ± . 460) . 

Argumentation classes demonstrates improvement (t = 6.92) in 
critical thinking ability ( average r = .129, var . = .012, jc = 5, 

N = 549, 95% C.I . ± . 107) . 

Participation in competitive forensics also demonstrated 
improvement in critical thinking across the investigations 
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( average r = .203, var . = .010, k = 8, N = 1577, 95% C.I. ± 

.152) . 

The results indicate that all methods of communication skill 
improvement generate gains in critical thinking. The largest 
effect, however, was observed for competitive forensic 
participation when compared to a public speaking class (d = .40) 
or an argumentation class (d = 1.13). 

The cross-sectional studies permitted a variety of 
comparisons. The first comparison considered when one group of 
participants has been part of a communication skill exercise and 
the change in critical thinking was compared to a control group. 
The average effect for this comparison demonstrated a larger gain 
in critical thinking ability for the communication skill exercise 
group ( average r = .241, var . = .025, k = 10, N = 1526, 95% C.I. 

± .270) . 

The comparison of competitive forensics to an argumentation 
or public speaking class demonstrates greater gain ( average r - 
.271, var . = .015, k = 5, N = 455, 95% C.I. ± .190) for 
competitive forensics participants. 

The participation in argumentation classes (or "enhanced" 
public speaking classes) when compared to normal public speaking 
classes finds argumentation classes generating greater 
improvements in critical thinking ( average r = .364, var . = .022, 
k = 5, N = 362, 95% C.I. ± .206). 
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Conclusions 

The average effects demonstrate that participation in public 
speaking, argumentation, and debate improves critical thinking 
skills. This is important because the findings illustrate that 
participation in public communication skill building exercises 
consistently improved critical thinking. Participation in 
forensics demonstrated consistently the largest improvement in 
critical thinking scores whether considering longitudinal or 
cross-sectional designs. 

Unfortunately this summary cannot adequately address the 
concerns about the reliance on the Watson-Glaser measure of 
critical thinking. While other measures of critical thinking 
[Bursack (1961) used Sarett's, "How Do You Think Test?", Douglas 
(1951) used Johnson's (1942) measure of reflective thinking. 

Green & Klug (1990) evaluated written essays, Howell (1961) used 
the California Analogies and Reasoning Test, Tame (1958) used a 
combination of Bradley's Logical Reasoning Test, Henning's 
Problem Recognition Test, and Sarret's measure] were utilized by 
investigations, the bulk of existing data relies on the Watson- 
Glaser measure (in all of its forms). This limitation requires 
additional research that both develops and utilizes alternative 
devices. However, while the investigations primarily relied on a 
single measurement instrument, a comparison to the other 
measurement efforts produced consistently and larger effects. 

That is, the other devices demonstrated improvement in critical 
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thinking for all the public communication skill efforts which 
argues for the generalizability of the conclusion. The 
importance of this observation is that while dependence on the 
Watson-Glaser instrument as the primary measurement device 
creates the ability to undermine the results should any flaw or 
limitation exist in the instrument, current alternative 
measurement devices generate the similar conclusions . Future 
research should consider whether instrumentation could be 
improved or alternative instruments to Watson-Glaser contain 
similar flaws. 

The impact of that improvement of critical thinking is 
illustrated in Table 5. Using Rosenthal's (1984) Binomial Effect 
Size Display (BESD) technique, a comparison can be made between 
those participating and those not participating in communication 
skill improvement programs/classes. Using BESD, the average 
effect (r = .18) indicates a 44% increase in the number of 
persons that score above the mean (the increase from 41% to 59%). 
This improvement in critical thinking ability indicates a large 
substantial positive influence of public communication training 
relating to critical thinking. 

One issue that the forensics community needs to consider is 
the integration of the activity into the practices of 
departmental instruction. The central issue regarding 
communication, as a field, involves the fact that the process is 
an action necessary to living. Our field is a lived experience 
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that every person enacts each day. Regardless of the area 
(organizational, interpersonal, rhetorical, mass, technological, 
small group, etc.,), there exists an aspect of performance and/or 
competence within that area which is a part of the ongoing 
theoretical and research tradition. One result, in the cross- 
sectional comparison, demonstrates the effect of public speaking 
may be improved by incorporating more aspects of argumentation 
into the curriculum. 

The results demonstrate the value of forensics participation 
in improving critical thinking. The granting of academic credit 
for participation in forensics appears justified. The effects 
also point to the additional advantage that debate and forensics 
participation can provide to training solely in public speaking. 
The companion activities of engaging in both argument and 
counterargument, whether required in public speaking, discussion, 
argumentation, and/or forensics competition better prepare 
students to become full participants in society. This evidence 
provides a partial answer to charges of "anti-humanism" in the 
last few years (Roth, 1996) . 

This paper provides some evidence for meeting the calls of 
those for evidence of educational accountability. Forensic 
participation (as well as other forms of public communication 
instruction) can be justified on the basis of the critical 
thinking improvement offered. Forensics participation 
demonstrated larger effects for improvement in all conditions. 
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particularly those calling for a comparison of the methods of 
instruction. These results provide important evidence to support 
the maintenance of forensics programs in an era of increased 
educational accountability, downsizing, and budgetary cutbacks. 

Public communication (forensics, argumentation, public 
speaking) , as a part of this larger endeavor, has a contribution 
to make both to the research, theory, and practice. The 
forensics community needs to contribute to the issues regarding 
how to improve the practice. While knowing that participation 
improves critical thinking is beneficial, the communication 
community needs to consider how to serve the participants using 
better and develop more effective approaches. Knowing that our 
contributions are positive should encourage confidence in our 
ability to experiment and to evaluate. The challenge is to 
integrate these experiences as a part of our overall curriculum 
rather than a view public communication skill training as a 
separate component. After all, improving critical thinking 
skills should benefit all students, regardless of the major or 
which class they next enroll. 

Future research should consider the impact of public address 
skills courses on the ability to listen and organize. One would 
expect that forensics participation (in all its competitive and 
classroom forms) would generate positive outcomes along a whole 
host of skills. The ability to listen, organize, and take notes 
all provide a basis for success in a variety of endeavors. 
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Future effort should target the particular activities that 
stimulate students to develop critical skills. The current 
results provide evidence for the effectiveness of the techniques 
but contribute little to the explanation of those effects. 

The impact of public communication training on the critical 
thinking ability of the participants is demonstrably positive. 
This summary of existing research reaffirms what many exdebaters 
and others in forensics, public speaking, mock trial, or 
argumentation would support the conclusion: participation 
improves the thinking of those involved! 
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FOOTNOTES 



-*-A comparison of this method with other methods of meta- 



analysis (Johnson, Mullen, & Salas, 1995) demonstrates that this 
method differs from methods used by Hedges and Olkin or 
Rosenthal. The article demonstrates little difference on the 
question of mean effect as demonstrated in Figure 1 (p. 101) or 

ti 

on issues of variability in Figure 3 (p. 103). The differences 
are found in the use of significance testing of the mean effect 
(see Figure 2, p. 102) and moderator prediction (see Figure 4, 
pl04) . The difference is that the Hunter and Schmidt technique 
does not improve in power as the sample size or number of studies 
increases. The purpose of the Hunter-Schmidt procedure was to 
develop a technique that was invariant to sample size, therefore 
it is consistent with the design of the technique to differ from 
methods that increase in power as sample size increases. 
Consequently, the estimates in this report are conservative with 
regard to interpretations of statistical significance. Hunter 
and Schmidt recommend that the use of the significance test be 
discontinued and reliance and confidence intervals be 
substituted. This report follows that recommendation. 
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Table 1 

List of Studies with Associated Statistical and Other Information 



Study 


Date 


Allen 


1995 


Baumgartner 


1965 


Beckman 


1956 


Beihl 


1963 


Brembeck 


1947 


Bursack 


1961 


Colbert 


1987 


Colbert 


1995 


Cross 


1971 


Douglas 


1951 


Frank 


1967 


Green 


1990 


Howell 


1942 


Hurst 


1961 


Jackson 


1961 


Ness 


1967 


Smith 


1942 


Tame 


1958 


Whalen 


1991 



Pre/Post 
r N 



.092 


.096 


104 


.354 


.380 


34 


.066 


.066 


293 


.083 


.092 


106 


. 196 


.196 


202 


.217 


.275 


328 


.303 


.329 


285 


.005 


.006 


148 


.072 


.078 


136 


.392 


.426 


11 


.105 


.111 


103 


. 671 


.671 


11 


.207 


.225 


213 


.169 


.180 


87 


.118 


.126 


123 


.172 


. 180 


164 


.164 


. 164 


309 



vs . Control 



r 


r' 


N 


.210 


.229 


138 


.286 


.296 


77 



.136 


.136 


413 


.342 


.373 


285 



.512 


.557 


20 


.429 


.451 


206 


.449 


.449 


23 


.051 


.055 


415 


.343 


.367 


157 


.172 


.179 


249 


.045- 


.049 


331 


. 143 


. 155 


34 


.458 


.475 


49 



Measure 

Watson-Glaser 

Watson-Glaser 

Watson-Glaser 

Watson-Glaser 

Watson-Glaser 

Sarret Test 

Watson-Glaser 

Watson-Glaser 

Watson-Glaser 

Combination 

Watson-Glaser 

Graded Essays 

Watson-Glaser 

California 

Watson-Glaser 

Watson-Glaser 

Watson-Glaser 

Combination 

Watson-Glaser 
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Note . The designation "r" indicates the uncorrected effect. The 
designation "r ' " indicates the effect after correcting for error 
due to attenuated measurement, restriction in range, and 
regression to the mean. All averages are calculated using the 
effects after they have been corrected for the appropriate 
errors . 
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Table 2 

List of Effects by Type of Longitudinal Comparison 



Study Date 

Allen 1995 



Baumgartner 


1965 


Beckman 


1956 


Beihl 


1963 


Brembeck 


1947 


Bursack 


1961 


Colbert 


1987 


Colbert 


1995 


Cross 


1971 


Douglas 


1951 


Frank 


1967 


Green 


1990 


Howell 


1942 


Hurst 


1961 


Jackson 


1961 


Ness 


1967 


Tame 


1958 



Pre to Post Change 



r 


r ' 


N 


-.035 


-.038 


37 


.041 


.044 


32 


.257 


.285 


35 


.354 


.380 


34 


.066 


.066 


293 


.083 


.092 


106 


.196 


.196 


202 


.217 


.275 


328 


.303 


.329 


285 


. 005 


.006 


148 


. 072 


.078 


136 


.392 


.426 


11 


.105 


.111 


103 


. 671 


. 671 


11 


.207 


.225 


213 


.169 


.180 


87 


. 118 


.126 


123 


. 172 


.180 


164 


. 164 


.164 


309 



Type of Experience 
Public Speaking 
Argumentation Class 
Forensics 
Public Speaking 
Argumentation Class 
Public Speaking 
Argumentation Class 
Forensics 
Forensics 
Forensics 
Forensics 

Argumentation Class 
Public Speaking 
Argumentation Class 
Forensics 
Public Speaking 
Forensics 
Public Speaking 
Forensics 
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34 



Table 3 

List of Study Effects and Coding when Using cross-sectional 
Designs 



Study 


Date 


r 


r' 


N 


Type of Comparison 


Allen 


1995 


. 133 


.145 


71 


Control/Public Speaking 






. 115 


.125 


69 


Public Speaking/Argumentation 






.432 


.470 


72 


Forensics /Public Speaking 






.245 


.266 


66 


Control /Argumentation 






.327 


.355 


67 


Forensics /Argumentation 






. 486 


.528 


69 


Control /Forensics 


Baumgartner 


1965 


.286 


.296 


77 


Control/Public Speaking 


Brembeck 


1947 


. 136 


.136 


413 


Control/Argumentation 


Colbert 


1995 


.342 


.373 


285 


Control /Forensics 


Douglas 


1951 


.512 


.557 


20 


Public Speaking/PS + A 


Frank 


1967 


. 429 


.451 


206 


Public Speaking/PS + A 


Green 


1990 


.449 


.449 


23 


Control /Forensics 


Howell 


1942 


.051 


.055 


415 


Control /Forensics 


Hurst 


1961 


.343 


.376 


157 


Control/Public Speaking 


Jackson 


1961 


.172 


.179 


249 


Public Speaking/Forensics 


Ness 


1967 


.099 


.113 


227 


Control/Public Speaking 


Smith 


1942 


.143 


.155 


34 


Public Speaking/PS + A 


Whalen 


1991 


.168 


.185 


34 


Argumentation/A + F 






.440 


.458 


32 


Public Speaking/A + F 






.402 


.417 


33 


Public Speaking/Argumentation 



0 
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Table 4 



Summary of 


Effect 










Type 


of Design 




Cross Sectional 


Longitudinal 


average r 


. 176 




.196 


variance 


.010 




.028 


k 


17 




13 


N 


2657 




2395 


95% Cl 


. 124 




.291 






Type of Measurement 




Cross Sectional 


Longitudinal 




Watson-Glaser 


Other 


Watson-Glaser Other 


average r 


. 160 


.215 


.192 .226 


variance 


.010 


.007 


.029 .013 


k 


12 


5 


10 3 


N 


1875 


782 


2105 290 


95% Cl 


. 124 


. 150 


.303 .123 




Effects when 


Comparing 


Types of Experience 




Cross Sectional 


Longitudinal 




n 

CO 

< 

< 


A/PS 






C A or 


PS v. PS PS A F 


average r 


.241 .271 


.364 


.145 .129 .203 



O 




variance 


.025 


015 


.022 


.066 


.012 


.010 


k 


10 


5 


5 


6 


5 


8 


N 


1526 


455 


362 


531 


549 


1577 


95% Cl 


.270 


190 


.206 


.460, 


. 107 


. 152 


Note. For 


type of skill 


comparison, CS = 


Communication 


Skill, 


= Public 


Speaking Class, 


C = 


Control, A 


= Argumentation 


Class , 



and F = Forensics 
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Table 5 

Binomial Effect Size Display for Effects 



Assuming a correlation of . 18 



Percentage of Persons 
Scoring on Critical Thinking 

Below the Mean Above the Mean 

Involved in Communication 41% 59% 

Not Involved 59% 41% 
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